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ABSTRACT 

The National Elevator Industry Educational Program 
(NEIEP) is a training trust established by an industry trade 
association and a union to maintain the supply of well-trained 
competent elevator mechanics. The NEIEP conducts Train the Trainer 
seminars that present a skill-building program to give each 
instructor a range of techniques and strategies for teaching and 
learning. A qualitative evaluation of the effectiveness of one of the 
seminars was conducted, A Level 1 (introductory) seminar was 
evaluated by participants at a Level 2 (more advanced) seminar about 
a year later. Participants indicated the impact that the initial 
seminar had on their performance as an instructor. Data from the 
evaluation were classified according to competency area and then 
presented to participants on the second day of the Level 2 seminar. 
The purpose was to illustrate techniques of formative evaluation for 
the seminar participants. Several inferences about the impact of the 
Level 1 seminar are possible. Students learned and put into practice 
some new planning and instructional techniques. They retained an 
acceptable amount of information about adult learning and applied it 
to some degree, and they learned and used evaluation techniques. The 
Level 1 seminar appears successful in providing basic cognitive 
knowledge and promoting performance change. Five tables summarize 
evaluation questions and responses. (SLD) 
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Train the Trainer Seminar Qualitative Evaluation 

The National Elevator Industry Educational Program (N.E.I.E.P.) is a training trust 
created as part of a collective bargaining benefit package between the National Elevator 
Industries, Inc. (N.E.I.I.)^ an Industry Trade Association representing most of the 
nation's elevator manufacturing and service companies, and the International Union of 
Elevator Constmctors (I.U.E.C.). N.E.I.E.P., also called the Program, is responsible for 
the design, development, and evaluation of courses if instruction and alternative 
learning experiences for classroom and home study programs in the construction, 
service, maintenance and modernization of elevators, escalators, and dumb waiters for 
Helpers (apprentice) and Mechanics journeymen) members of the union. The stated 
goal of the Program is to maintain the supply of well-trained, competent elevator 
mechanics for the industry. Pursuant to this end, N.E.I.E.P. offers Train the Trainer 
Seminars to field subject matter experts (SMFs) to instruct in their respective Local's 
N.E.LE.P. Training school. 

Instructors in the Program have a great deal of technical knowledge about the 
industry. They also have a smcere interest in helping new helpers prepare to perform as 
competent elevator mechanics. However, they often assume this instructional role with 
little or no formal teaching preparation or experience. 

In 1980, N.E.LE.P. conmiissioned a project to design, develop, implement and 
evaluate a series of seminars appropriate for training instructors in basic and advanced 
instructional techniques. A national needs assessment was conducted in order to collect 
specific data concerning the strengths and weaknesses of the instructional program. 
Based on the results of these acti\dties, a Level I seminar was designed and conducted 
for field instructors dealing with basic instructional skills and techniques. 

A Level I Seminar is a two day introductory learning experience that introduces the 
SME to a variety of teaching/ learning concepts identified as eight Competency Areas. 
These Competency Areas consist of instmction on thirteen topics: one. Learning 
Theory and Practice- The adult Learner; two. Planning for instruction- Instructional 
Analysis; three. Management of the Instmctional Process- The Process of Instruction 
and Selecting and Evaluation Media for Instruction; four. Presentation Techniques- 
Utilizing Media for Instruction, Promoting Learner Participation and 
Designing/Producing Instructional Materials; five. Motivation- Basic Motivation 
Techniques; six. Strategies for Teaching Techniques; seven. Communication- Effective 
Speaking Techniques and Eliciting Learning Feedback; and eight, Evaluation- 
Instructor/Learner Assessment Techniques. The Seminar is designed to be conducted 
in fifteen instructional hours. The pace is rapid. Emphasis of this seminar is on 
instructors acquiring the basic skills to successfully conduct the basic teaching activities 
required of N.E.LE.P. instructors throughout the United States; that is, training 
individuals to be capable and competent elevator mechanics. The purpose of this 
seminar is not to tell instructors how to teach, but to present a skill-building program 



1 



that provides each instructor with a range of basic and advanced techniques, strategies 
and applications that may be considered for any number of teaching and learning 
situations. 

For this to occur, teaching in the NEIEP program should be viewed as 
facilitating the student's use of learning resources. These resources may be an 
instructor's knowledge of specific elevator-related subject matter or that of 
other elevator professionals, contributions from students or specific materials 
such as manufacturer's guides, models, slides and so forth. A good NEIEP 
instructor establishes an environment where students can interact with 
him/her, each other and the subject matter in an atmosphere that allows both 
the students and instructor to work together toward a common goal. That is, 
training individuals to be capable and competent elevator mechanics. 
(Patterson & DiPaolo, 1981, p.2) 

Most of the areas v^thin this seminar are addressed on an application basis with 
technical and theoretical explanation kept to a minimum. Emphasis is placed on ways 
of directly transferring learning from the seminar to the actual instructional setting of 
each instructor. 

After SME's are identified and selected by their respective Local to become 
instructors, they are invited to attend a Level I, Training the Instructor Seminar. This 
seminar program is unique for three reasons. First, the program is designed on careful 
analysis of instructor needs on a national basis, not upon any single individual or 
regional group. Second, the program is specifically designed for the instructor whose 
needs vary in complexity and detail from other industry groups having teaching/learning 
problems. Third, these presentations do not follow any single theory or approach as the 
only way ti conduct a seminar or to acquire instructional skills. The format is basic and 
practical and provides each instructor with a foundation upon which he can add his own 
personal preferences and style. 

The formal evaluation given at the conclusion of the Level I Seminar is a participant 
reaction Likert Semantic Differential instrument soliciting responses to perceptions of 
general performance. This type of data is typical of the data collected at similar 
seminars and workshops. Table 1 is a compilation of the percentage of responses for 
the 1990/91 instructional season. While the instrument is of value in determining the 
acceptance of the seminar content, presenters and support materials and services, it is 
does not gather data in regard to effectiveness in eliciting behavioral change over time. 
Question 10 is an open-ended request for pertinent information. Table 2 is a random 
selection of typical statements taken from two consecutive seminars. Again, while these 
data may be important for formative evaluation of seminar elements, they do not 
measure seminar effectiveness on instructors in the instructional setting. Over the 
years, the data have been remarkably consistent in ratings and comments. 



Table 1 

SEMINAR EVALUATION RESPONSES BY PERCENTAGE 

Seminar Participant Reaction Evaluation Instrument 

1. During this seminar I learned: 

[100] a great deal [ 0] some things [ 0] nothing 

2. Given my previous background as a NEIEP instnactor, most of this material was: 

[3] advanced [97] about right [ 0] too basic 

3. The instruction in the seminar was paced: 

[7] too slow [83] fast [10] about right [0] slow [0] too slow 

4. Generally, the seminar instructional activities were: 

[99] clear [ 1] adequate [ 0] confusing 

5. As a NEIEP instructor, this seminar: 

[89] met my needs [1 1] met some of my needs [ 0] didn't meet my needs 

6. Overall, I would rate the seminar materials (workbook, resource book, 

transparencies, films, etc.) as: 

[92] excellent [ 8] good [ 0 ] fair [0] poor 

7. Overall, I rate the performance of the seminar leaders as: 

[99] excellent [l]good [Ojfair [0]poor 
8. 1 believe my attending this seminar was: 

[100] worthwhile [ 0] somewhat useful [ 0] a waste of my time 
9. Overall, I would rate this seminar as: 

[91] excellent [ 9] good [ 0 ] fair [ 0] poor 

lO.On the back, list any comments about the seminar facilities, lodging, meals, 
consultant performance, in-class and homework activities, pre-seminar materials, 
evaluations, etc. 

(Torrence, 1991, pps.4,6,8) 



Table 2 

SELECTION OF TYPICAL OPEN-ENDED EVALUATION STATEMENTS 



Seminar Evaluation Instrument Open-end Responses 

• I found this seminar excellent in every aspect- thank you 

• I commend the organizers of this seminar for giving me the nev/ tools of confidence I 
will be bringing to my own classroom. It would be great if we could extend this to 
learn even more. 

• Very fast paced, lots of information. 

« I learned most at the end of the seminar. The materials are good, but in no special 
order. I would like a good index. I would like more time spent on how to overcome 
speech problems. Mainly the spring board entrance. 

• For the first time it was a little overlearning, but it all fell into place. Thank you for 
asking me here. 

• I have a better understanding of NEIEP's operation. I can use this information to 
help my students. 

• Need more time, Dave's input was helpful, Hotel was great. Need more choice in 
meals; and more time to prepare. To get an honest evaluation on technic [sic. 
technique]. Video tape was more fun than I expected. 

• Although the pace was fast, after a period of time, it was clear that the use of 
Mechanical equipment is a advantage for the teacher. 

• Everything was fine. 

• Found lighting inadequate 

• Very good job, glad you let me attend. 

• Tlie room is too cold, poor lighting, no choice of meals. Keep Amos and Andy - 
excellent teaching 'TEAM". 

• Amos and Andy Great. Keep Them!! 

• Overall I enjoyed the Seminar. I just hope I apply what I learned. 

• Facility was good in most all respects. Dave, Amos, and Andy were all very 
articulate. All did an excellent job of presenting the seminar material. In addition 
all three were very nice, as v/ere all the seminar participants. EXCELLENT 
SEMINAR!! (Torrence, 1991, pps.5,7,9-10) 
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Field reports of improvement in the instructional quality of the program over the 
years has been attributable to the seminar program. Other factors such as youth, 
enthusiasm and a more dynamic industry could account for some of the seminar 
acceptance but has never been measured as a weighted factor. While the immediate 
seminar evaluation is consistently positive, there is evidence of cognitive "overload". 

Obviously, a meaningful measure of the effectiveness of the Level I Seminar would 
be a rigorous measure of change in instructor performance with some form of pre and 
post seminar performance checklist. This form of data is rarely generated. A report on 
forty-five Fortune 500 companies in 1988 showed that while 100 percent of the 
companies employed participant reaction forms in their evaluation, only 30 percent 
used measures of learning and only 15 percent used measures of behaviors 
(Brandenburg & Schultz, 1988), Given the national scope and the size of the staff, this 
type of quantitative evaluation is not feasible. 

Level I is the first of three Training the Instructor seminars. To reinforce existing 
concepts and to further develop certain areas of persistent concern that was recorded, a 
Level II Train the Trainer seminar was developed and has been offered for the past sk 
years. Level II is offered to instructors who continue in the Program during the second 
year of instruction beyond the Level I Seminar. This gave the Program an opportunity 
to address all of the instructors in an informal, supportive environment to perform a 
qualitative evaluation of the impact of the Level I seminar. 

Qualitative evaluation differs with quantitative evaluation in philosophic 
orientation. Quantitative evaluation conforms to the rigors of experimental and 
quasi-experimental educational and psychological research methodology such as that 
expounded by Campbell and Stanley. The quantitative paradigm emphasizes the testing 
of hypotheses. Often, this requires very large numbers to provide meaningful statistics. 

Quantitative research emphasizes standardizatiofiy precision, objectivity, and 
reliability of measurement as well as replicability and generalizability of 
findings. Thus, quantitative research is characterized not only by a focus on 
producing numbers but on generating numbers which are suitable for 
statistical tests. (Schofield & Anderson, 1984, pps.8-9) 

Most educational evaluators view it as the ideal. Strict adherence to this methodology 
often, however, can preclude the opportunity to gather meaningful data. 

Qualitative evaluation has been gaining in general acceptance since the 1970's as an 
alternative or parallel paradigm. The dichotomy of the two paradigms has been 
described as the distinction between the "naturalistic" and "scientific" paradigms. 
Qualitative methodology utilizes subjective measures. 




Qualitative methods consist of three types of data collection: (1) in-depth, 
open-ended interviews; (2) direct observation; and (3) written documents, 
including such sources as open-ended written items on questionnaires, 
personal diaries, and program records. (Patton, 1987, p.7) 

Qualitative and naturalistic methods were largely shunned by educational evaluators 
prior to the 70's as producing unacceptably "soft" data. / nalysis methods for qualitative 
data usually involve some form of analytic induction. Analysis depends upon the nature 
of the data and the conceptual framework employed in the analysis. 

[Qualitative inquiry] (a) is conducted in natural settings, such as schools or 
neighborhoods; (b) utilizes the researcher as the chief ^Hnstrumen^^ in both 
data-gathering and analysis, * * *(c) emphasizes ^Hhick description^^ that is, 
obtaining ^ear, ^^ricK\ '^deep\ data which illuminate everyday patterns of 
action and meaning from the perspective of those being studied * * .(d) tends 
to focus on social processes rather than primarily or exclusively on outcomes, 
(e) employs multiple data-gathering methods, especially observation and 
interviews, and (f) uses and inductive approach to data analysis, extracting its 
concepts form the mass of particular detail which constitutes the data base . . 
. . .(Schofield & Anderson, 1984, p.9) 

A qualitative evaluation of the seminar effectiveness in measuring change was feasible. 

Potential problems that evaluators may encounter in collecting evaluation 
information fall into two categories: one, problems of cooperation or access to data; and 
two, technical problems in data collection. The data collection and analysis in this 
evaluation was from data amassed at the Level II seminar and corroborated by student 
evaluations that were made available to the evaluator, instructor interviews, and 
interviews of Local Joint Educational Program administrators. 

As part of the the "warm-up*' exercise at the begimiing of the Level II seminar, the 
question is posed as each participant exchanges his/her name and some personal vita: 
What is the most important thing that you now do as a result of attending the Level I 
seminar? As each responds, the data is recorded or a flip chart for posting and 
comparison. Each instructor responds to only one performance change. Each response 
reflects the single most important change in his/her repertory of instructional 
techniques as recalled from the Level I seminar. These statements of performance 
change have been collected over time. Each statement has been assigned to one of the 
competency areas and tabulated. Table 3 is a summary of the responses with their 
assignment to the appropriate competency area and the percent In that the data were 
analyzed by this reporter, the inter-rater reliability factor is null. After competency area 
assignment was made, the data were reviewed by the consultants for corroboration. No 
differences in assignment were noted. Table 4 is a summary of the competency areas by 
percent of responses. 



Tables 

SUMMARY OF SEMINAR RESPONSES 
WITH COMPETENCY AREA ASSIGNMENT 



Response Competency Area 

• Using visuals 4 

• Using overhead projector 4 

• Better planning 2 
« Sensitive to personal appearance (hands in pocket) 7 

• More eye contact 5 

• Better preparation 2 

• Better lesson plans 2 

• Write objectives and use it as focus 3 

• Advance preparation 2 

• More class participation 3 

• Use of "hand on" bring ins 4 

• Prepare lesson plan 2 

• Hand-on materials 4 

• Organize Class seating ("U" shape) 3 

• Use overhead 4 

• Use flip chart 4 

• Use of visual aids 4 

• Use of Springboards 2 

• More class participation 3 

• Application of videos 4 

• Better classroom layout 3 

• Better use of media, especially videos 4 




Table 3 Cont. 

SUMMARY OF SEMINAR RESPONSES WITH 
COMPETENCY AREA ASSIGNMENT 



Response Competency Area 

• Ways to connect with students 5 

• Classroom control 3 

• Making a lesson plan 2 

• Set objectives 2 

• Get student involvement 4 

• Use of flipchart 4 

• Ask rflore questions 6 

• Use more media: overhead, flipchart, etc. 4 

• Bring in "hands on" 4 

• More use of overhead 4 

• More discussion/less lecture 6 

• Use of visual aids 4 

• Better planning 2 

• Use of computers 4 

• Flipcharts and overhead 4 

• Smaller class 3 

• Use of classroom equipment and E- Lab 4 

• Lesson plans structure 2 

• Better questioning techniques 7 

• Use of overhead projector 4 

• Use of chalkboard 4 

• More group discussion 6 
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Table 3 Cont. 

SUMMARY OF SEMINAR RESPONSES WITH 
COMPETENCY AREA ASSIGNMENT 



Response Competency Area 

• Better class involvement 6 

• Observe students and draw them out 5 

• Focus on students 3 

• Creating visual materials 4 

Table 4 

SUMMARY OF COMPETENCY AREAS 
BY PERCENT OF RESPONSES 

Competency Area Percent of Responses 

1. Learning Theorj^ and Practice 0% 

2. Planning for instmction- Instructional Analysis 20.0% 

3. Management of the Instructional Process 16.7% 

4. Presentation Techniques 43.7% 

5. Motivation- Basic Motivation Techniques 6.3% 

6. Strate^es for Teaching Techniques 8.3% 

7. Communication 4.2% 

8. Evaluation 0% 



A review of the data required interpretation. Qualitative evaluation allows for 
flexibility in analysis. Competency area four, Presentation Techniques-Utilizing Media 
for Instruction, Promoting Learner Participation and Designing/Producing Instructional 
Materials accounted for almost one-half of the total responses. Much of this could be 
explained by the highly manipulative nature of this strategy. The most encouraging 
aspect of these responses was the diversity of the media forms reported as being 
employed. 
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All areas except areas one and eight had a reasonable impact on the instructors 
performance. It would appear that competency area one, Learning Theory and 
Practice- the adult Learner, and competencj^ area eight, Evaluation- Instructor/Learner 
Assessment Techniques generated no responses. Interviews with instructors uncovered 
probable explanations. 

Competency area one dealt with the concept of learning and with the psychology of 
adult learning. The competency area was basically a cognitive learning experience. Any 
understanding of adult learning requirements would be expressed as sensitivity to 
change in performance in the other areas. The change in other areas, however, could 
not be assumed to be the result of acting upon the cognitive learning of area one. The 
amount of retention of adult learning concepts had to be measured. 

Competency area eight deals with the concepts and practices of evaluation. The 
Program provides for summative evaluation through the use of Standardized Unit and 
Module examinations. These exams are carefully controlled and form part of necessary 
communications v^th the central administration. Therefore, any change in instructor 
performance would have to come from personal testimonies of novel uses of evaluation 
for formative evaluation. These personal testimonies had to be gathered. 

Each seminar has a pretest and a posttest to gather "summative" data regarding the 
mastery of content as well as a "formative" measure of the success in presenting the 
content. The Level II Pretest includes several questions dealing with cognitive concepts 
that were presented in Level I to review retention of presented information. The 
pretest is collected at the beginning of the seminar before any review of previous 
material is made. Several questions were used to gather information about the 
retention of cognitive concepts regarding learning theory, competency area 1; and about 
evaluation, competency area 8. Table 5 is a summary of the relevant questions and the 
pretest performance on each item. 

The high percentages of recall on those questions addressing cognitive information 
on adult learning suggests a functional knowledge of general adult learning psychology 
sufficient to sensitize the instructor to be amenable to performance change in the other 
competency areas. The acceptable distribution of responses over these areas suggests 
that this sensitivity is contributory. This suggests performance change in Competency 
Area one. Learning Theory and Practice- the Adult Learner. 

On the second day of the Level II seminar, a presentation to build upon the basic 
evaluation information presented in Level I is given. The purpose is tc develop 
formative evaluation techniques to help improve classroom instruction. To determine 
the impact of the Level I presentation, instructors are asked: "Other than giving the Unit 
and Final exams, what other ways do you determine if the students are learning the 
material?" Consistently, instructors acknowledge use of questioning techniques to 
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Table 5 

SUMMARY OF RELEVANT PRETEST QUESTIONS 
AND RESPONSE SUMMARIES 

Level II Pretest Questions 

7. Generally, adult learners are 
(98)*A. less easily discouraged 

( 1) B. less accepting of change 

( 1) C. less fearful of failure 

( 0) D. less self-conscious and cautious 

( 0) E. I don't know 

8. The NEIEP instructor can maximize trainees' feelings of success and interest by 

( 0) A. not disclosing the learning goals 

( 0) B. use difficult questions during discussion 
(92) *C. make instruction relevant to real world practice 
( 8) D. reward the trainees frequently 
( 0) E. I don't know 

9. A major problem encountered by adults learners returning to the classroom is 
( 0) A. poor eyesight 

(10) B. forgetting how to do math problems 

(89)*C unlearning mistaken knowledge and procedures 

( 1) D. lack of respect for the instructor 
( 0) E. I don't know 

* Indicates correct answer 
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Table 5 Cont. 

SUMMARY OF RELEVANT PRETEST QUESTIONS 
AND RESPONSE SUMMARIES 

Level II Pretest Questions 

14. Trainees learn most effectively when the instructor 
(10) A. models desired behaviors 

( 8) B. maintains a reasonable level of anxiety in the classroom 
( 4) C. criticizes behaviors, not people 

(77) *D. focuses upon learners' needs and problems 

( 1) E. I don't know 

28. A major rationale for NEIEP Instructors to conduct "self-evaluation" procedures is to 
(89)* A. evaluate and improve their own performance 
( 0) B. single out the problem trainees 

( 0) C. locate mistakes in the Modules 

( 9) D. give feedback to successful trainees 

( 2) E. I don't know 

50. Most adult learning is 

( 1) A. based on a single approach 

(91)*B. self-directed 
( 1) C. similar to adolescent learning 
( 6) D. based on guided discussion 
( 1) E. I don't know 

* Indicates correct answer (Torrence, 1991, pps.21-26) 



ERIC 



12 

15 



provide feedback. Also, when asked: "What do you do to evaluate your own teaching 
ability?" Statements of formative evaluation techniques Consistently include: 

• Video and audio taping of their classes 

• Giving ungraded "pop" quizzes 

• Creating their own unit exams 

• Allowing the students to evaluate the instructor. 

These responses suggest performance change in Competency Area eight, Evaluation- 
Instructor/Learner Assessment Techniques. 

Based upon the data, several inferences concerning the impact of the Level I 
seminar can be made. In that the data questionnaire eliciting responses of performance 
changes is limited to only one statement, it must be assumed that other changes take 
place in other categories. Based on the number of incidents of stated performance 
changes relating to competency areas 2 through 7, Planning for instruction- 
Instructional Analysis; Management of the Instructional Process- The Process of 
Instruction and Selecting and Evaluation Media for Instruction; Presentation 
Techniques- Utilizing Media for Instruction, Promoting Learner Participation and 
Designing/Producing Instructional Materials; Motivation- Basic Motivation Techniques; 
Strategies for Teaching Techniques; and Communication- Effective Speaking 
Techniques and Eliciting Learning Feedback; it can be inferred that: 

1. Students learned and put into practice to some degree various planning techniques, 

altered their management of the instmctional process, altered their presentation 
techniques, used motivation techniques, tried various strategies for teaching, and 
altered their communications style. 

Given the high degree of cognitive retention of material covered in Competency Area 
one. Learning Theory and Practice- the adult Learner, it can be inferred that: 

2. Students retained an acceptable amount of information regarding adult learning 

theory. 

Given the functional knowledge of general adult learning psychology sufficient to 
sensitize the instructor to be amenable to performance change in the other competency 
areas and the acceptable distribution of responses over these areas, suggesting that 
sensitivity is contributory, it can be inferred that: 

3. The instmctors learned and put into practice to some degree knowledge of adult 

learning theory. 
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Given the instructor feedback of employment of practices using evaluation techniques 
presented in Competency Area eight, Evaluation- Instructor/Learaer Assessment 
Techniques in the Lxvel I seminar to generate formative evaluation suggesting 
performance change, it can be infened that: 

4. The instructors learned and put into practice to some degree evaluation and 

assessment techniques. 

In that all the competency areas have been involved in cognitive retention and/or 
performance alteration, it can be inferred that: 

5. The Level I Train the Trainer seminar was successful as an intervention in providing 

functional basic cognitive information and promoting positive performance changes 
over time. 

While this methodology would not be considered as an experimental design in the 
traditional quantitative evaluation sense, it is valid empirical data hence an appropriate 
qualitative method to ascertain the relative effectiveness of the Train the Trainer 
intervention in order to make gross generalizations regarding retention or rejection. 
Qualitative inferences suggest positive functional performance of contract. 

Prepared and submitted by David R. Torrence, Ed.D., Staff Assistant for 
Instructional Development, 800/228-8220 ext.l5. 
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